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Claims 

1. A cooking-operation recognition system comprising: 

a sensing means for acquiring observation data which 
includes at least either of an image around the hands of 
a person who performs a cooking operation and an environment 
sound that is produced by the cooking operation; 

a feature -quantity template in which various cooking 
operations that are predetermined for various ingredients 
are stored in a table form, together with a template certainty 
factor that is a certainty factor which is predetermined 
for each cooking operation of the ingredients; 

a cooking-flow creating means for, based on the 
observation data which is acquired by the sensing means, 
calculating an observation certainty factor which indicates 
the certainty factor of at least either of an ingredient 
that is cooked by the person and the cooking operation of 
the ingredient , and based on this observation certainty factor , 
creating a cooking flow for a dish which is prepared by the 
person ; 

aprimary certainty-factor calculating means for, based 
on the template certainty factor and the observation certainty 
factor, calculating a primary certainty factor which is a 
certainty factor for each of objects that make up the cooking 
flow which is created by the cooking-flow creating means; 
and 
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a cooking-operation recognizing means for calculating 
a final certainty factor based on the primary certainty factor 
which is calculated by the primary certainty-factor 
calculating means, updating the feature-quantity template 
by considering this final certainty factor as a new template 
certainty factor , and based on this feature-quantity template , 
recognizing the ingredient and the cooking operation that 
correspond to the observation data which is acquired by the 
sensing means. 

2. The cooking-operation recognition system according 
to claim 1, further comprising: 

a cooking-flow database which stores cooking flows 
which are created in advance for various dishes; and 

a relevance-factor calculating means for, using flow 
matching, calculating a relevance factor that indicates which 
of the cooking flows that are stored in the recipe-flow database 
is closest to the cooking flow which is created by the 
cooking-flow creating means, 

the cooking-operation recognizing means recognizing 
an ingredient and a cooking operation, based on the relevance 
factor which is calculated by the relevance-factor calculating 
means and the primary certainty factor. 

3. The ingredient cooking-operation recognition system 
according to claim 1, further comprising: 
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an Ingredient table which Is formed by an Ingredient 
record that has a field where data that Indicates an Ingredient 
Is stored and a field where various kinds of data on the 
Ingredient are stored together with a template certainty 
factor that Is predetermined for the data; 

a cooked-food table which Is formed by a cooked-food 
record that has a field where data that Indicates a cooking 
operation Is stored and a field where various kinds of data 
on a change In an Ingredient by the cooking operation are 
stored together with a template certainty factor that Is 
predetermined for the data; 

a cooking-operation table which Is formed by a 
cooking-operation record that has a field where data that 
Indicates a cooking operation Is stored and a field where 
various kinds of data on an environment sound that Is produced 
by the cooking operation are stored together with a template 
certainty factor that Is predetermined for the data; 

a cooklng-reclpe database which stores data that 
Indicates an Ingredient which Is used for various cooking 
recipes and data that Indicates a cooking operation for the 
Ingredient ; 

a table storing means for storing at least any one 
of the tables and an extracting means for extracting all 
kinds of Ingredients and all kinds of cooking operations 
which are stored In the cooklng-reclpe database; and 

a feature-quantity template creating means for creating , 
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as a feature-quantity template, a table which includes, as 
items, each ingredient that is extracted by the extracting 
means and each cooking operation that is extracted by the 
extracting means, 

the feature-quantity template creating means writing, 
in an optional field of the fields which make up the 
feature-quantity template, as the template certainty factor, 
the greatest certainty factor of the template certainty factors 
which are stored in each field of at least any record of 
the ingredient record of the ingredient table that stores 
the data which indicates the ingredient that corresponds 
to the optional field, the cooked-food record of the cooked-food 
table that stores the data which indicates the cooking operation 
that corresponds to the optional field and the 
cooking-operation record of the cooking-operation table that 
stores the data which indicates the cooking operation that 
corresponds to the optional field. 

4. The ingredient cooking-operation recognition system 

according to any one of claims 1 to 3, characterized in that: 
the sensing means has a thermal camera and an optical 
camera; and 

an image which is acquired by the sensing means includes 
a thermal image which is photographed by the thermal camera 
and an optical image which is photographed by the optical 
camera . 
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5. The Ingredient cooking-operation recognition system 
according to claim 4, characterized in that the cooking-flow 
creating means: removes a background region from the optical 
image by executing a background difference processing on 
the optical image; removes, from the thermal image, as a 
background region, a region where the temperature is equal 
to, or higher than, a predetermined value; calculates an 
ingredient region which indicates an Ingredient by multiplying 
the thermal image whose background region is removed and 
the optical image whose background region is removed; and 
calculates an observation certainty factor based on this 
ingredient region . 

6. The ingredient cooking-operation recognition system 
according to any one of claims 1 to 5, characterized in that 
the cooking-flow creating means obtains a histogram of hue 
and a histogram of saturation from an image which is acquired 
by the sensing means, and estimates an ingredient by obtaining 
a correlation between these histograms and a color- feature 
template which is predetermined for each Ingredient. 

7. The ingredient cooking-operation recognition system 
according to any one of claims 1 to 6 , characterized in that; 

the relevance-factor calculating means estimates a 
cooking recipe biased on a relevance factor; and 



56 



PI 5 89 (translation) 



a guidance means is further provided for giving, to 
the person, guidance on the cooking recipe which is estimated 
by the relevance-factor calculating means. 

8. An ingredient cooking-operation recognition program, 

characterized by allowing a computer to function as : 

a sensing means for acquiring observation data which 
includes at least either of an image around the hands of 
a person who performs a cooking operation and an environment 
sound that is produced by the cooking operation; 

a feature-quantity template in which various cooking 
operations that are predetermined for various ingredients 
are stored in a table form, together with a template certainty 
factor that is a certainty factor which is predetermined 
for each cooking operation of the ingredients; 

a cooking-flow creating means for, based on the 
observation data which is acquired by the sensing means, 
calculating an observation certainty factor which indicates 
the certainty factor of at least either of an ingredient 
that is cooked by the person and the cooking operation of 
the ingredient , and based on this observation certainty factor , 
creating a cooking flow for a dish which is prepared by the 
person; 

aprimarycertainty-f actor calculatingmeans for, based 
on the template certainty factor and the observation certainty 
factor, calculating a primary certainty factor which is a 
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certainty factor for each of objects that make up the cooking 
flow which is created by the cooking-flow creating means; 
and 

a cooking-operation recognizing means for calculating 
a final certainty factor based on the primary certainty factor 
which is calculated by the primary certainty-factor 
calculating means, updating the feature-quantity template 
by considering this final certainty factor as a new template 
certainty factor , and based on this feature-quantity template , 
recognizing the ingredient and the cooking operation that 
correspond to the observation data which is acquired by the 
sensing means. 
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